Disparity image stitching and visualization method based on multiple pairs of binocular cameras

ABSTRACT

The present invention discloses a disparity image stitching and visualization method based on multiple pairs of binocular cameras. A calibration algorithm is used to solve the positional relationship between binocular cameras, and the prior information is used to solve a homography matrix between images; internal parameters and external parameters of the cameras are used to perform camera coordinate system transformation of depth images; the graph cut algorithm has high time complexity and depends on the number of nodes in a graph; the present invention divides the images into layers, and solutions are obtained layer by layer and iterated; then the homography matrix is used to perform image coordinate system transformation of the depth images, and a stitching seam is synthesized to realize seamless panoramic depth image stitching; and finally, depth information of a disparity image is superimposed on a visible light image.

TECHNICAL FIELD

The present invention belongs to the field of image processing andcomputer vision, and particularly relates to a method comprising thesteps of calculating a homography matrix between images through externalparameters (a rotation vector R and a translation vector T) betweencameras, finding an optimal stitching seam between images by graph cut,using R, T, the homography matrix and an optimal transition area tostitch disparity images, and finally fusing the disparity images andvisible light images for display.

BACKGROUND

At present, driverless technology is developing rapidly, and safetyneeds to be ensured for driverless technology. However, by simply usingvisible light images, it is not possible to well locate an obstacle,obtain the distance of the obstacle, and accurately locate the vehicleof one's own. With the improvement of the technology for obtainingdisparity images based on binocular cameras, disparity images are alsoused as basic data in the field of driverless technology. However,limited by the accuracy of the disparity images, the field angle of thebinocular cameras is small, and a single pair of binocular camerascannot provide sufficient environmental information for the vehicle ofone's own. The larger the field angle of a vehicle is, the more completethe information obtained will be, and the higher the guarantee ofdriving safety will be. In order to make the disparity images have awider range of field angle, it is necessary to stitch multiple disparityimages together. Currently, the following two methods are mainly usedfor stitching disparity images:

1. Stitching Method Using Feature Points

This method is to extract feature matching points between images, thensolve a rotation vector R and a translation vector T between cameras,and finally stitch disparity images according to R and T. The advantagesof this method are that the stitching effect is relatively good, the useis flexible, and the method can be used in most scenes; and thedisadvantages are that the calculation complexity is high and the methodcannot meet the high real-time requirements of driverless technology.

2. Stitching Method Using Camera Calibration

This method is to obtain external parameters R and T between cameras byusing checkers, and then stitch disparity images. This method has asmall amount of stitching calculation and high real-time performance,but it is easy to produce stitching seams during a disparity imagestitching process, which makes the stitching effect poor.

The disparity image stitching process is divided into two processes:camera coordinate transformation and image coordinate transformation.The transformation of a camera coordinate system requires the use ofinternal parameters K of cameras and external parameters RT betweencameras to calculate in a three-dimensional coordinate system; and thetransformation of an image coordinate system requires the use of ahomography matrix H between camera images and an optimal transition areaof visible light images for stitching. An image coordinate systemtransformation process requires pre-registration, and it takes a lot oftime to calculate the external parameters and the homography matrixbetween cameras by matching feature points. Experiments show that cameraangles are fixed, the positional relationship RT between cameras and theinternal parameters K of cameras can be calculated by a calibrationalgorithm, and the homography matrix between two images can be derivedthrough RT and the internal parameters K of cameras, and through therelationship between a global coordinate system and an image coordinatesystem, so that the feature point matching time is omitted by the priorinformation. Image registration is completed and solutions are obtainedby a graph cut algorithm. As the graph cut algorithm is time-consuming,in order to achieve real-time performance, the images need to beprocessed layer by layer to reduce the calculation complexity of graphcut. An optimal stitching seam obtained based on the images is used toseamlessly stitch the disparity images after image coordinate systemtransformation. Finally, disparity image information is superimposed onvisible light images to facilitate the observation of the depthinformation of the environment.

SUMMARY

To overcome the defects in the prior art, the present invention providesa disparity image stitching and visualization method based on multiplepairs of binocular cameras: a homography matrix between images ispre-solved based on the prior information (i.e., the positionalrelationship R and T between cameras), the traditional graph cutalgorithm is improved to increase the efficiency of the graph cutalgorithm and then is used for stitching disparity images, and thedisparity images are fused with visible light images to make it moreconvenient to observe the depth of the environment. A stitching processrequires image information and depth image information obtained by eachbinocular camera.

The present invention has the following specific technical solution:

A disparity image stitching and visualization method based on multiplepairs of binocular cameras, comprising the following steps:

Step 1): calibrating internal parameters and external parameters of eachbinocular camera; the internal parameters K include a focal length focusand optical center coordinates C_(x), C_(y); the external parametersinclude a rotation matrix R and a translation vector T; obtaining abaseline length baseline of each binocular camera by calibration; andobtaining visible light images and disparity images of two binocularcameras;

Step 2): calculating homography matrix: calculating a homography matrixH according to the internal parameters and external parameters of eachbinocular camera, the placement angle between the cameras, and a sceneplane distance d; selecting an appropriate value of d according toactual conditions, with the value range thereof being 8-15 in ingeneral; and the specific steps are as follows:

2-1) Imaging a plane in a scene by two binocular cameras, and assumingthat the unit normal vector of the plane in the coordinate system of thefirst binocular camera is N, and the distance from the plane to thecenter (coordinate origin) of the first binocular camera (i.e., thescene plane distance) is d, then the plane n is expressed as:

N ^(T) C ₁ =d  (1)

Wherein C₁ is the three-dimensional coordinate of a three-dimensionalpoint P in the coordinate system of the first binocular camera, and thecoordinate of the three-dimensional point P in the coordinate system ofthe second binocular camera is C₂, then the relationship between C₁ andC₂ is:

C ₂ =RC ₁ +T  (2)

Formula (2) is further expressed as:

$\begin{matrix}{C_{2} = {{{RC}_{1} + {T\frac{1}{d}N^{T}C_{1}}} = {{\left( {R + {T\frac{1}{d}N^{T}}} \right)C_{1}} = {H^{\prime}C_{1}}}}} & (3)\end{matrix}$

Wherein R and T are respectively a rotation vector and a translationvector from the first binocular camera to the second binocular camera;

2-2) Transforming C₁ and C₂ in step 2-1) from the internal parameters ofthe cameras into an image coordinate system:

c ₁ =K ₁ C ₁  (4)

c ₂ =K ₂ C ₂  (5)

It can be obtained from formulas (3), (4) and (5) that:

$\begin{matrix}{c_{2} = {{{K_{1}\left( {R + {T\frac{1}{d}N^{T}}} \right)}{K_{2}}^{- 1}c_{1}} = {K_{1}H^{\prime}{K_{2}}^{- 1}c_{1}}}} & (6)\end{matrix}$

Finally, a calculation formula of the homography matrix calculated bythe internal parameters and the external parameters is obtained:

$\begin{matrix}{H = {{K_{1}H^{\prime}{K_{2}}^{- 1}} = {{K_{1}\left( {R + {T\frac{1}{d}N^{T}}} \right)}{K_{2}}^{- 1}}}} & (7) \\{H = \begin{bmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{bmatrix}} & (8)\end{matrix}$

Wherein c₁ is a corresponding coordinate of C₁ in the coordinate systemof an imaging plane, and c₂ is a corresponding coordinate of C₂ in thecoordinate system of the imaging plane; K₁ is the internal parameters ofthe first binocular camera; K₂ is the internal parameters of the secondbinocular camera; and the finally obtained transformation matrix H is a3*3 matrix, and a₁₁-a₃₃ represent specific values.

Step 3): using the internal parameters of the binocular cameras and theexternal parameters between the binocular cameras obtained in step 1)and step 2) to perform camera coordinate system transformation of thedisparity images; and the specific steps are as follows:

3-1) Using the internal parameters K₁ (i.e., the baseline lengthbaseline₁ and the focal length focus₁) of the first binocular camera torestore the disparity images to a point cloud in the coordinate systemof the first binocular camera, and the calculation formulas of thethree-dimensional coordinates C₁ (X₁, Y₁, Z₁) of the point cloud are asfollows:

$\begin{matrix}{Z_{1} = \frac{{baseline}_{1}*{focus}_{1}}{{disparity}_{1}}} & (9) \\{X_{1} = \frac{\left( {x_{1} - C_{x}} \right)*{baseline}_{1}}{{disparity}_{1}}} & (10) \\{Y_{1} = \frac{\left( {y_{1} - C_{y}} \right)*{focus}_{1}}{{disparity}_{1}}} & (11)\end{matrix}$

Wherein x₁ and y₁ are the pixel coordinates of the first binocularcamera; disparity is a disparity value;

3-2) Using the external parameters R and T from the first binocularcamera to the second binocular camera to transform the camera coordinatesystem of the point cloud and obtain the three-dimensional coordinatesof the point cloud in the coordinate system of the second binocularcamera; and the coordinate transformation formula is as follows:

$\begin{matrix}{\begin{pmatrix}X_{2} \\Y_{2} \\Z_{2}\end{pmatrix} = {{R\begin{pmatrix}X_{1} \\Y_{1} \\Z_{1}\end{pmatrix}} + T}} & (12)\end{matrix}$

3-3) Using the internal parameters K₂ (i.e., the baseline lengthbaseline₂ and the focal length focus₂) of the second binocular camera torestore the point cloud to the disparity images, at this time, only Z₂is needed to calculate the disparity value in the coordinate system ofthe second binocular camera, and the calculation formula is as follows:

$\begin{matrix}{{disparity}_{2} = \frac{{baseline}_{2}*{focus}_{2}}{Z_{2}}} & (13)\end{matrix}$

Step 4): building overlapping area model: using the homography matrix Hbetween images obtained in step 2) to calculate an overlapping area ROIof images, and modeling the overlapping area; and the specific steps ofbuilding a mathematical model are as follows:

4-1) For the pixels of two images in the overlapping area, calculatingthe second norm of the RGB pixels corresponding to the overlapping areaof the two images, and building t-links; the calculation formulas of thesecond norm are as follows:

e(p,q)=∥p—p′∥+∥q−q′∥  (14)

∥p−p′∥=(R _(p) −R _(p′))²+(G _(p) −G _(p′))²+(B _(p) −B _(p′))²  (15)

∥q−q′∥=(R _(q) −R _(q′))²+(G _(q) −G _(q′))²+(B _(q) −B _(q′))²  (16)

Wherein e(⋅) is a weight function, p is a source image, q is a targetimage, p is the pixel value of one point in the image p, p′ is the pixelvalue of a p adjacent point, q is the pixel value of one point in thetarget image, q′ is the pixel value of a q adjacent point, R_(p) is thevalue of R channel at p point, R_(p′) is the value of R channel at p′point, G_(p) is the value of G channel at p point, G_(p′) is the valueof G channel at p′ point, B_(p) is the value of B channel at p point,B_(p′), is the value of B channel at p′ point, R_(q) is the value of Rchannel at q point, R_(q′) is the value of R channel at q′ point, G_(q)is the value of G channel at q point, G_(q′) is the value of G channelat q′ point, B_(q) is the value of B channel at q point, and B_(q′) isthe value of B channel at q′ point;

4-2) Finding an optimal stitching line for the built model, and solving(the stitching seam) by graph cut; an energy function is defined as:

E(f)=Σ_(p,q∈N) S _(p,q)(l _(p) ,l _(qi))+Σ_(p∈P) D _(p)(I _(p))  (17)

Wherein S_(p,q) is a smoothing term representing the cost of assigning apair of pixels (p, q) in the overlapping area to (l_(p), l_(q)), l_(p)is a label assigned to the pixel p, l_(q) is a label assigned to thepixel q, and D_(P) is a data term representing the cost of marking thepixel p in the overlapping area as l_(p);

Step 5): dividing each image into blocks with a size of B₁*B₂, takingthe divided blocks as nodes of a graph, performing graph cut to find alocal optimal solution, then continuing to divide each nodecorresponding to an optimal stitching line corresponding to B₁*B₂ untila final block size is equal to a pixel value, and finally andapproximately finding a global optimal solution by finding the localoptimal solution each time;

Step 6): using the homography matrix H to perform image coordinatesystem transformation of the disparity images; seamlessly stitching theoptimal stitching line in step 5); and the specific steps of disparityimage stitching are as follows:

6-1) Transforming the disparity image of the first binocular camera intothe image coordinate system of the second binocular camera:

$\begin{matrix}{{w\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix}} = {H\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}} & (18)\end{matrix}$

Wherein x₁ and y₁ are the coordinates in the image coordinate system ofthe first binocular camera; x₂ and y₂ are the coordinates in the imagecoordinate system of the second binocular camera; and w is anormalization coefficient;

6-2) Stitching image: comparing the positions of the first binocularimage after image coordinate system transformation and the secondbinocular image corresponding to an optimal stitching seam, and mergingtwo visible light images and two disparity images respectively;

When the number of binocular cameras is greater than two, repeatingsteps 3)-6) to obtain disparity images with a larger field angle.

Step 7): adding the stitched disparity information to the visible lightimages; and the specific steps are as follows:

7-1) Converting the disparity images to color images, replacingdisparity information with color information, and using different colorsto represent different depths;

7-2) Superimposing and fusing the color images obtained from thedisparity images and the visible light images, and the superpositionmethod is a weighted average method:

Fused image=k*visible light image+(1−k)*color image  (19)

Wherein k is a weight coefficient.

The present invention has the following beneficial effects: the presentinvention realizes large-field-angle panoramic disparity image display;the algorithm of the present invention achieves real-time performance,and realizes large-disparity seamless panoramic disparity imagestitching and visualization. The present invention has the followingadvantages: (1) the program has low requirements on memory and hardware,and can achieve real-time performance on Nvidia TX2; (2) the program issimple and easy to implement; (3) after obtained, the prior informationcan be directly passed in as parameters to be used as default values;(4) the optimal stitching seam obtained from the images is applied todisparity image stitching to achieve seamless stitching; and (5)disparity image information is superimposed on visible light images.

The present invention makes full use of the prior information of theimages and reduces the time of image registration. The method proposedhas good scalability; panoramic display of multiple pairs of cameras canbe realized by simply inputting R, T and internal parameters K ofcameras, and manually setting d value; and the disparity imageinformation is superimposed on the visible light images to display thedepth information of the environment more intuitively.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a system structure diagram of binocular cameras of anembodiment of the present invention.

DETAILED DESCRIPTION

The present invention proposes a disparity image stitching andvisualization method based on multiple pairs of binocular cameras, andwill be described in detail below in combination with drawings andembodiments.

The present invention uses multiple pairs of horizontally placedbinocular cameras as an imaging system to perform multi-viewpoint imagecollection, wherein K₁ is the internal parameters of the first binocularcamera, and K₂ is the internal parameters of the second binocularcamera. The resolution of each binocular camera is 1024*768, the videoframe rate is greater than 20 frames per second, and a system referencestructure is shown in FIG. 2. The spatial transformation relationship Rand T between each pair of binocular cameras is calculated on thisbasis, and the homography matrix H between images is calculated throughR, T and the distance d of the imaging plane; the horizontal translationof each image is calculated by taking an intermediate image as abenchmark; and finally, the calculated parameters are used as inputs forstitching and visualization. The specific process is as follows:

1) System calibration and data collection

1-1) Calibrating each pair of binocular cameras to obtain the internalparameters including focal length and optical center and the externalparameters including rotation and translation of each pair of binocularcameras;

1-2) Connecting each pair of binocular cameras to multiple computers,and using a router to control and conduct synchronous data collection;

1-3) Using special customized calibration boards to collect images atthe same time; paying attention to ensure that the positionalrelationship between the binocular cameras is consistent during thecollection process and keep the calibration boards fixed; and rotatingthe calibration boards to collect 10-15 groups of images for each pairof binocular cameras according to the actual conditions.

2) Calculating homography matrix between image transformations

2-1) Imaging a plane in a scene by two cameras, and assuming that theunit normal vector of the plane in the coordinate system of the firstcamera is N, and the distance from the plane to the center (coordinateorigin) of the first camera is d, then the plane π can be expressed as:

N ^(T) C ₁ =d

Wherein C₁ is the coordinate of a three-dimensional point P in thecoordinate system of the first camera, X₁ and the coordinate of thethree-dimensional point P in the coordinate system of the second camerais C₂, then the relationship between the two is:

C₂ = R * C₁ + T$C_{2} = {{{RC}_{1} + {T\frac{1}{d}N^{T}C_{1}}} = {{\left( {R + {T\frac{1}{d}N^{T}}} \right)C_{1}} = {H^{\prime}C_{1}}}}$

2-2) Obtaining the homography matrix obtained in step 2-1) from thecoordinate system of the first camera, and the homography matrix needsto be transformed into the coordinate system of the imaging plane:

c ₁ =K ₁ C ₁

c ₂ =K ₂ C ₂

H=K ₁ H′K ₂ ⁻¹

The value of d in the above formula can be set manually, and the restare fixed values. In this way, the homography matrix H from the firstbinocular camera to the second binocular camera is obtained.

3) Using the internal parameters of the binocular cameras and theexternal parameters between the binocular cameras obtained in steps 1)and 2) to perform camera coordinate system transformation of thedisparity images;

3-1) Using the internal parameters K₁, etc. of the first pair ofbinocular cameras to restore the disparity images to a point cloud inthe coordinate system of the first camera:

$Z_{1} = \frac{{baseline}_{1}*{focus}_{1}}{{disparity}_{1}}$$X_{1} = \frac{\left( {x_{1} - C_{x}} \right)*{baseline}_{1}}{{disparity}_{1}}$$Y_{1} = \frac{\left( {y_{1} - C_{y}} \right)*{focus}_{1}}{{disparity}_{1}}$

3-2) Using R and T from the first binocular camera to the secondbinocular camera to transform the camera coordinate system of the pointcloud:

$\begin{pmatrix}X_{2} \\Y_{2} \\Z_{2}\end{pmatrix} = {{R\begin{pmatrix}X_{1} \\Y_{1} \\Z_{1}\end{pmatrix}} + T}$

Using the internal parameters K₂ of an intermediate viewpoint binocularcamera to restore the point cloud to the disparity images, at this time,only Z₂ is needed to obtain the disparity images, and the calculationformula is as follows:

${disparity}_{2} = \frac{{baseline}_{2}*{focus}_{2}}{Z_{2}}$

Calculating overlapping area of images and solving optimal stitchingseam by modeling: first, calculating an overlapping area ROI through thehomography matrix between images, and then building an overlapping areamodel; and the specific steps are as follows:

4-1) Calculating the size of the overlapping area by the homographymatrix between images:

Taking the four vertices (0, 0), (img.cols, 0), (img.cols, img.rows) and(0, img.rows) of an image, calculating the transformed coordinates, thetransformed upper left corner coordinate is for the stitched image, andthe homography transformation matrix H is:

$H = \begin{bmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{bmatrix}$

The calculation formulas are:

$x = {\frac{x^{\prime}}{w^{\prime}} = \frac{{a_{11}u} + {a_{12}v} + a_{13}}{{a_{31}u} + {a_{32}v} + a_{33}}}$$y = {\frac{y^{\prime}}{w^{\prime}} = \frac{{a_{21}u} + {a_{22}u} + a_{23}}{{a_{31}u} + {a_{32}u} + a_{33}}}$

Wherein x is the x-coordinate of the source image p point afterperspective transformation, y is the y-coordinate of the source image ppoint after perspective transformation, u is the x-coordinate of thesource image p point, and v is the y-coordinate of the source image ppoint;

4-2) Building an energy model (seam-driven image stitching), andconstructing an energy function of a graph cut algorithm:

${E(1)} = {{\sum\limits_{p \in P}{D_{p}\left( l_{p} \right)}} + {\sum\limits_{{({p,q})} \in N}{S_{p,q}\left( {l_{p},l_{q}} \right)}}}$

Wherein the data term D_(p)(l_(p)) represents the assigned value ofpixel p in the overlapping area:

$\left\{ {\begin{matrix}{{D_{p}(1)} = {{0\mspace{14mu}{D_{p}(0)}} = \mu}} & {{{if}\mspace{14mu} p} \in {{\partial I_{0}}\bigcap{\partial P}}} \\{{D_{p}(0)} = {{0\mspace{14mu}{D_{p}(1)}} = \mu}} & {{{if}\mspace{14mu} p} \in {{\partial I_{1}}\bigcap{\partial P}}} \\{{D_{p}(0)} = {{D_{p}(1)} = 0}} & {otherwise}\end{matrix}\quad} \right.$

In order to avoid marking errors, pi is set to be a very large number;

S_(p,q)(l_(p), l_(q)) is a smoothing term;

S _(p,q)(l _(p) ,l _(q))=I _(*)(p)+I _(*)(q)

I _(*)(p)=∥I ₀(⋅)−I ₁(⋅)∥₂

5) After the model is built, obtaining a solution by graph cut, and theresult is an optimal stitching seam. It can be known that theconstruction of the energy function is very important for the result ofthe stitching seam.

5-1) As the operation time of graph cut is related to the number ofnodes in a graph, and the algorithm complexity is relatively high, onlyby down-sampling or stratifying the overlapping area to reduce thenumber of nodes in the constructed graph, and making the local optimalsolution obtained by this method be approximately equal to the globaloptimal solution, can the real-time performance of the algorithm meetrequirements.

5-2) In addition, the parallelization of the graph cut algorithm canalso achieve a further acceleration effect. (Fast graphcut on GPUCVPR2008) 6) The specific steps of disparity image stitching are asfollows:

6-1) Transforming the depth image of the first binocular camera into theimage coordinate system of the second binocular camera:

${W\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix}} = {H\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}$

6-2) Stitching image: comparing the positions of the disparity imageafter image coordinate system transformation and an intermediatedisparity image corresponding to the optimal stitching seam, and mergingthe two disparity images.

Completing the disparity image stitching of one pair of binocularcameras by steps 1)-6), and repeating steps 1)-6) to complete thedisparity image stitching of the second pair of binocular cameras (e.g.,the second and third binocular cameras).

7) Adding the stitched disparity information to the visible lightimages:

7-1) Converting the disparity images to color images, replacingdisparity information with color information, and using different colorsto represent different depths, wherein the color images calculated fromthe disparity images includes but is not limited to pseudo-color imagesand rainbow images;

7-2) Superimposing and fusing the color images obtained from thedisparity images and the visible light images, and the superpositionmethod is a weighted average method:

Fused image=k*visible light image+(1−k)*color image

k is a weight coefficient; when k is relatively large (1-*0.5), visiblelight information can be observed more clearly; and when k is relativelysmall (0.5-0), more depth information can be observed.

1. A disparity image stitching and visualization method based onmultiple pairs of binocular cameras, comprising the following steps:step 1): calibrating internal parameters and external parameters of eachbinocular camera; the internal parameters K include a focal length focusand optical center coordinates C_(x), C_(y); the external parametersinclude a rotation matrix R and a translation vector T; obtaining abaseline length baseline of each binocular camera by calibration; andobtaining visible light images and disparity images of two binocularcameras; step 2): calculating homography matrix: calculating ahomography matrix H according to the internal parameters and externalparameters of each binocular camera, the placement angle between thecameras, and a scene plane distance d; and the value range of d is 8-15m; step 3): using the internal parameters of the binocular cameras andthe external parameters between the binocular cameras obtained in step1) and step 2) to perform camera coordinate system transformation of thedisparity images; step 4): building overlapping area model: using thehomography matrix H between images obtained in step 2) to calculate anoverlapping area ROI of images, and modeling the overlapping area; step5): dividing each image into blocks with a size of B₁*B₂, taking thedivided blocks as nodes of a graph, performing graph cut to find a localoptimal solution, then continuing to divide each node corresponding toan optimal stitching line corresponding to B₁*B₂ until a final blocksize is equal to a pixel value, and finally and approximately finding aglobal optimal solution by finding the local optimal solution each time;step 6): using the homography matrix H to perform image coordinatesystem transformation of the disparity images; seamlessly stitching theoptimal stitching line in step 5); when the number of binocular camerasis greater than two, repeating steps 3)-6) to obtain disparity imageswith a larger field angle; and step 7): adding the stitched disparityinformation to the visible light images.
 2. The disparity imagestitching and visualization method based on multiple pairs of binocularcameras according to claim 1, wherein the specific steps of calculatinghomography matrix described in step 2) are as follows: 2-1) imaging aplane in a scene by two binocular cameras, and assuming that the unitnormal vector of the plane in the coordinate system of the firstbinocular camera is N, and the distance from the plane to the center ofthe first binocular camera (i.e., the scene plane distance) is d, thenthe plane π is expressed as:N ^(T) =C ₁ =d  (1) wherein C₁ is the three-dimensional coordinate of athree-dimensional point P in the coordinate system of the firstbinocular camera, and the coordinate of the three-dimensional point P inthe coordinate system of the second binocular camera is C₂, then therelationship between C₁ and C₂ is:C ₂ =RC ₁ +T  (2) formula (2) is further expressed as: $\begin{matrix}{C_{2} = {{{RC}_{1} + {T\frac{1}{d}N^{T}C_{1}}} = {{\left( {R + {T\frac{1}{d}N^{T}}} \right)C_{1}} = {H^{\prime}C_{1}}}}} & (3)\end{matrix}$ wherein R and T are respectively a rotation vector and atranslation vector from the first binocular camera to the secondbinocular camera; 2-2) transforming C₁ and C₂ in step 2-1) from theinternal parameters of the cameras into the image coordinate system:c ₁ =K ₁ C ₁  (4)c ₂ =K ₂ C ₂  (5) it can be obtained from formulas (3), (4) and (5)that: $\begin{matrix}{c_{2} = {{{K_{1}\left( {R + {T\frac{1}{d}N^{T}}} \right)}{K_{2}}^{- 1}c_{1}} = {K_{1}H^{\prime}{K_{2}}^{- 1}c_{1}}}} & (6)\end{matrix}$ finally, a calculation formula of the homography matrixcalculated by the internal parameters and the external parameters isobtained: $\begin{matrix}{H = {{K_{1}H^{\prime}{K_{2}}^{- 1}} = {{K_{1}\left( {R + {T\frac{1}{d}N^{T}}} \right)}{K_{2}}^{- 1}}}} & (7) \\{H = \begin{bmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{bmatrix}} & (8)\end{matrix}$ wherein c₁ is a corresponding coordinate of C₁ in thecoordinate system of an imaging plane, and c₂ is a correspondingcoordinate of C₂ in the coordinate system of the imaging plane; K₁ isthe internal parameters of the first binocular camera; K₂ is theinternal parameters of the second binocular camera; and the finallyobtained transformation matrix H is a 3*3 matrix, and a₁₁-a₃₃ representspecific values.
 3. A disparity image stitching and visualization methodbased on multiple pairs of binocular cameras according to claim 1,wherein the specific steps of step 3) are as follows: 3-1) using theinternal parameters K₁ (i.e., the baseline length baseline₁ and thefocal length focus₁) of the first binocular camera to restore thedisparity images to a point cloud in the coordinate system of the firstbinocular camera, and the calculation formulas of the three-dimensionalcoordinates C₁ (X₁, Y₁, Z₁) of the point cloud are as follows:$\begin{matrix}{Z_{1} = \frac{{baseline}_{1}*{focus}_{1}}{{disparity}_{1}}} & (9) \\{X_{1} = \frac{\left( {x_{1} - C_{x}} \right)*{baseline}_{1}}{{disparity}_{1}}} & (10) \\{Y_{1} = \frac{\left( {y_{1} - C_{y}} \right)*{focus}_{1}}{{disparity}_{1}}} & (11)\end{matrix}$ wherein x₁ and y₁ are the pixel coordinates of the firstbinocular camera; disparity is a disparity value; 3-2) using theexternal parameters R and T from the first binocular camera to thesecond binocular camera to transform the camera coordinate system of thepoint cloud and obtain the three-dimensional coordinates of the pointcloud in the coordinate system of the second binocular camera; and thecoordinate transformation formula is as follows: $\begin{matrix}{\begin{pmatrix}X_{2} \\Y_{2} \\Z_{2}\end{pmatrix} = {{R\begin{pmatrix}X_{1} \\Y_{1} \\Z_{1}\end{pmatrix}} + T}} & (12)\end{matrix}$ 3-3) using the internal parameters K₂ (i.e., the baselinelength baseline₂ and the focal length focus₂) of the second binocularcamera to restore the point cloud to the disparity images, at this time,only Z₂ is needed to calculate the disparity value in the coordinatesystem of the second binocular camera, and the calculation formula is asfollows: $\begin{matrix}{{disparity}_{2} = {\frac{{baseline}_{2}*{focus}_{2}}{Z_{2}}.}} & (13)\end{matrix}$
 4. The disparity image stitching and visualization methodbased on multiple pairs of binocular cameras according to claim 1,wherein the specific steps of building overlapping area model describedin step 4) are as follows: 4-1) for the pixels of two images in theoverlapping area, calculating the second norm of the RGB pixelscorresponding to the overlapping area of the two images, and buildingt-links; the calculation formulas of the second norm are as follows:e(p,q)=∥p—p′∥+∥q−q′∥  (14)∥p−p′∥=(R _(p) −R _(p′))²+(G _(p) −G _(p′))²+(B _(p) −B _(p′))²  (15)∥q−q′∥=(R _(q) −R _(q′))²+(G _(q) −G _(q′))²+(B _(q) −B _(q′))²  (16)wherein e(⋅) is a weight function, p is a source image, q is a targetimage, p is the pixel value of one point in the image p, p′ is the pixelvalue of a p adjacent point, q is the pixel value of one point in thetarget image, q′ is the pixel value of a q adjacent point, R_(p) is thevalue of R channel at p point, R_(p′) is the value of R channel at p′point, G_(p) is the value of G channel at p point, G_(p′) is the valueof G channel at p′ point,; is the value of B channel at p point, B_(p′)is the value of B channel at p′ point, R_(q) is the value of R channelat q point, R_(q′) is the value of R channel at q′ point, G_(q) is thevalue of G channel at q point, G_(q′) is the value of G channel at q′point, B_(q) is the value of B channel at q point, and B_(q′) is thevalue of B channel at q′ point; 4-2) finding the optimal stitching linefor the built model, and solving the stitching seam by graph cut; anenergy function is defined as:E(f)=Σ_(p,q∈N) S _(p,q)(l _(p) ,l _(q))+Σ_(p∈p) D _(p)(l _(p))  (17)wherein S_(p,q) is a smoothing term representing the cost of assigning apair of pixels (p, q) in the overlapping area to (l_(p),l_(q)), l_(p) isa label assigned to the pixel p, l_(q) is a label assigned to the pixelq, and D_(P) is a data term representing the cost of marking the pixel pin the overlapping area as l_(p).
 5. The disparity image stitching andvisualization method based on multiple pairs of binocular camerasaccording to claim 3, wherein the specific steps of building overlappingarea model described in step 4) are as follows: 4-1) for the pixels oftwo images in the overlapping area, calculating the second norm of theRGB pixels corresponding to the overlapping area of the two images, andbuilding t-links; the calculation formulas of the second norm are asfollows:e(p,q)=∥p—p′∥+∥q−q′∥  (14)∥p−p′∥=(R _(p) −R _(p′))²+(G _(p) −G _(p′))²+(B _(p) −B _(p′))²  (15)∥q−q′∥=(R _(q) −R _(q′))²+(G _(q) −G _(q′))²+(B _(q) −B _(q′))²  (16)wherein e(⋅) is a weight function, p is a source image, q is a targetimage, p is the pixel value of one point in the image p, p′ is the pixelvalue of a p adjacent point, q is the pixel value of one point in thetarget image, q′ is the pixel value of a q adjacent point, R_(p) is thevalue of R channel at p point, R_(p′) is the value of R channel at p′point, G_(p) is the value of G channel at p point, G_(p′) is the valueof G channel at p′ point, B_(p) is the value of B channel at p point,B_(p′) is the value of B channel at p′ point, R_(q) is the value of Rchannel at q point, R_(q′) is the value of R channel at q′ point, G_(q)is the value of G channel at q point, G_(q′) is the value of G channelat q′ point, B_(q) is the value of B channel at q point, and B_(q′) isthe value of B channel at q′ point; 4-2) finding the optimal stitchingline for the built model, and solving the stitching seam by graph cut;an energy function is defined as:E(f)=Σ_(p,q∈N) S _(p,q)(l _(p) ,l _(q))+Σ_(p∈P) D _(p)(l _(p))  (17)wherein S_(p,q) is a smoothing term representing the cost of assigning apair of pixels (p, q) in the overlapping area to (l_(p),l_(q)), l_(p) isa label assigned to the pixel p, l_(q) is a label assigned to the pixelq, and D_(p) is a data term representing the cost of marking the pixel pin the overlapping area as l_(p).
 6. The disparity image stitching andvisualization method based on multiple pairs of binocular camerasaccording to claim 1, wherein the specific steps of disparity imagestitching described in step 6) are as follows: 6-1) transforming thedisparity image of the first binocular camera into the image coordinatesystem of the second binocular camera: $\begin{matrix}{{W\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix}} = {H\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}} & (18)\end{matrix}$ wherein x₁ and y₁ are the coordinates in the imagecoordinate system of the first binocular camera; x₂ and y₂ are thecoordinates in the image coordinate system of the second binocularcamera; and w is a normalization coefficient; 6-2) stitching image:comparing the positions of the first binocular image after imagecoordinate system transformation and the second binocular imagecorresponding to an optimal stitching seam, and merging two visiblelight images and two disparity images respectively.
 7. The disparityimage stitching and visualization method based on multiple pairs ofbinocular cameras according to claim 3, wherein the specific steps ofdisparity image stitching described in step 6) are as follows: 6-1)transforming the disparity image of the first binocular camera into theimage coordinate system of the second binocular camera: $\begin{matrix}{{W\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix}} = {H\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}} & (18)\end{matrix}$ wherein x₁ and y₁ are the coordinates in the imagecoordinate system of the first binocular camera; x₂ and y₂ are thecoordinates in the image coordinate system of the second binocularcamera; and w is a normalization coefficient; 6-2) stitching image:comparing the positions of the first binocular image after imagecoordinate system transformation and the second binocular imagecorresponding to an optimal stitching seam, and merging two visiblelight images and two disparity images respectively.
 8. The disparityimage stitching and visualization method based on multiple pairs ofbinocular cameras according to claim 4, wherein the specific steps ofdisparity image stitching described in step 6) are as follows: 6-1)transforming the disparity image of the first binocular camera into theimage coordinate system of the second binocular camera: $\begin{matrix}{{W\begin{pmatrix}x_{2} \\y_{2} \\1\end{pmatrix}} = {H\begin{pmatrix}x_{1} \\y_{1} \\1\end{pmatrix}}} & (18)\end{matrix}$ wherein x₁ and y₁ are the coordinates in the imagecoordinate system of the first binocular camera; x₂ and y₂ are thecoordinates in the image coordinate system of the second binocularcamera; and w is a normalization coefficient; 6-2) stitching image:comparing the positions of the first binocular image after imagecoordinate system transformation and the second binocular imagecorresponding to an optimal stitching seam, and merging two visiblelight images and two disparity images respectively.
 9. The disparityimage stitching and visualization method based on multiple pairs ofbinocular cameras according to claim 1, wherein the specific steps ofstep 7) are as follows: 7-1) converting the disparity images to colorimages, replacing disparity information with color information, andusing different colors to represent different depths; 7-2) superimposingand fusing the color images obtained from the disparity images and thevisible light images, and the superposition method is a weighted averagemethod:Fused image=k*visible light image+(1−k)*color image  (19) wherein k is aweight coefficient.
 10. The disparity image stitching and visualizationmethod based on multiple pairs of binocular cameras according to claim6, wherein the specific steps of step 7) are as follows: 7-1) convertingthe disparity images to color images, replacing disparity informationwith color information, and using different colors to representdifferent depths; 7-2) superimposing and fusing the color imagesobtained from the disparity images and the visible light images, and thesuperposition method is a weighted average method:Fused image=k*visible light image+(1−k)*color image  (19) wherein k is aweight coefficient.